Image processing method and apparatus, computer program, and computer-readable storage medium

ABSTRACT

This invention can perform a developing process or editing process, with a high response, for raw image obtained by image sensing by a digital camera or the like. When, therefore, raw image data is input, an extraction area in the raw image data is determined on the basis of the size of an image display area in which a processed image is displayed and a magnification set at this time. A developing process is then performed for partial raw image data in the determined area, and the resultant data is displayed in the image display area.

This is a continuation of U.S. patent application Ser. No. 11/048,756,filed Feb. 3, 2005, now pending.

FIELD OF THE INVENTION

The present invention relates to image processing for raw image dataobtained by image sensing by a digital camera, digital video camera, orthe like.

BACKGROUND OF THE INVENTION

When image sensing is performed by a digital still camera, a digitalvideo camera, or the like, the image data input from an imaging sensorsuch as a CCD or a CMOS sensor, is influenced by a developing process(luminance adjustment, color tone adjustment, sharpness processing, andthe like). The resultant data is then JPEG-Baseline coded and stored asa file in a predetermined storage medium (memory card) (for example,Japanese Patent Laid-Open No. 3-49483).

This coding, however, is lossy coding, and hence accompanies the lossesof some information. That is, there is a limit to the readjustment of aJPEG image file on a personal computer; this coding does not allow fineadjustment often requested by experimented users.

In order to meet such demands, there may be provided a technique ofstoring unprocessed image data in a storage medium, immediately after itis obtained by an imaging sensor, without any change or upon losslesscompression, and performing, on a PC, a process similar to a developingprocess performed in a digital camera. Such unprocessed image data isgenerally called raw image data. Performing a developing process forthis raw image data on the PC makes it possible to perform elaboratedeveloping so as to meet the demand of the power users.

When, however, a developing process for raw image data is to be done onthe PC, it is necessary to perform pixel interpolation processing andwhite balance processing in accordance with light source information atthe time of imaging, and luminance adjustment processing, color toneadjustment processing, and sharpness processing in accordance withconditions set by an operator, e.g., an exposure time, shutter speed,and zoom position, and the like.

The above developing process must be done for each pixel. In recentdigital cameras, the number of pixels of sensed images is generallyseveral millions or exceed ten million, and hence it requires much timeto obtain the result of a developing process after it is started.

As conceivable techniques for solving the above problem, the presentinventor has considered the following techniques.

The first technique is to perform image processing upon decimatingpixels and display a reduced image or reproduce a low-resolution imageprocessing result.

The second technique is to limit the display area, perform imageprocessing for only part of the image data which can be displayed withinthe display area, and reproduce the image processing result.

The third technique is to hold the image processing result, on anoverall image, in a storage medium such as a memory and use it forreproduction.

The fourth technique is to hold a plurality of image quality adjustmentprocessing results of an overall image in a storage medium such as amemory and use them for the reproduction and the display of processingresults after image quality adjustments.

According to the first technique, an increase in the number of decimatedpixels will shorten the processing time until reproduction. However, asthe number of decimated pixels increases, images with lower resolutionsare displayed. Such images are therefore not suitable for eithersharpness processing evaluation, or noise reduction processing, and thelike.

According to the second technique, since an image is partly processed,the overall image cannot be checked. This technique is therefore notsuitable for evaluating the overall color tone after white balanceprocessing or color adjustments. In addition, limiting a display areawill eliminate the degree of freedom of the user's operation.

The third technique requires a lot of time for the first reproductionand/or processing. In addition, when image quality adjustment isperformed, the image processing needs to be executed again. Each suchoperation is time-consuming.

The fourth technique requires much time for the first reproductionand/or processing. This technique may be effective when the number oftypes of parameters that can be used for image quality adjustment issmall. However, since there is a tendency towards diversification ofimage quality adjustments, even processing results which the user doesnot require may be held, resulting in an increase in computational load.

Furthermore, with improvements in image quality, image processing tendsto become more complex. In addition, the number of pixels constitutingan image tends to increase. It is therefore necessary to speed upreproduction and image processing.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the aboveproblems, and has as its object to provide a technique which allows theexecution of processing with a high response during a developing orediting process of raw image data sensed by a digital camera or thelike.

In order to achieve the above object, an image processing apparatusaccording to the present invention has, for example, the followingarrangement.

There is provided an image processing apparatus which develops and editsraw image data obtained by an image sensing device having an imagingsensor, comprising:

input means for inputting raw image data;

parameter setting means for setting an image processing parameter;

area size setting means for setting a size of an output area forreproduction/outputting of an image;

magnification setting means for setting a magnification of an imageoutput to the output area;

extraction area size determining means for determining a size of anextraction area in the raw image data on the basis of the size of theoutput area set by the area size setting means and the magnification setby the magnification setting means;

image processing means for performing image processing for partial rawimage data in the extraction area determined by the extraction area sizedetermining means, on the basis of the parameter set by the parametersetting means; and output means for outputting the image data obtainedby the image processing means to the output area.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the arrangement of a digital stillcamera as an image input device;

FIG. 2 is a flowchart showing the steps up to recording an imageobtained by the image input device;

FIG. 3 is a view showing the structure of an image file;

FIGS. 4A to 4C are views for explaining the pixel array of an imagesignal;

FIG. 5 is a flowchart showing the steps up to loading a raw image from arecording device and displaying it on an output device;

FIGS. 6A to 6C are views showing examples of the output results obtainedat the respective display magnifications set in an image processingapparatus;

FIGS. 7A to 7G are views showing examples of the output results obtainedat the respective display magnifications and the sizes of an imagedisplay area which are set in the image processing apparatus;

FIG. 8 is a flowchart showing the steps of the first means for changingimage information to be processed in accordance with informationassociated with display in the image processing apparatus;

FIG. 9 is a flowchart showing the steps of the process of displaying alow-resolution image until an image processing result corresponding toinformation associated with display is output;

FIG. 10 is a flowchart showing the steps of the process of switchingbetween displaying a processing result on low-resolution imageinformation and not displaying it in accordance with changed imagequality adjustment information;

FIG. 11 is a block diagram showing the arrangement of an apparatus whichperforms image processing according to the embodiment;

FIG. 12 is a view showing an example of a GUI window of an imageprocessing application in the embodiment; and

FIG. 13 is a flowchart showing the contents of processing to be done atthe time of storage processing in the embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments of the present invention will be described in detailbelow with reference to the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing the arrangement of a digital stillcamera serving as an image input device according to this embodiment.Referring to FIG. 1, reference numeral 101 denotes a photographing lens;102, a shutter/stop; 103, a photoelectric conversion device (to bereferred to as a CCD hereinafter) which converts an optical image of anobject (to be photographed) imaged by a photographing optical systeminto an electric signal; 104, an analog/digital converter (to bereferred to as an A/D converter) which converts an output from the CCD103 into a digital signal; 105, a digital processor which processes thesignal digitized by the A/D converter 104; 106, a CPU which performscamera control and signal processing; 107, a rewritable memory which isused for temporary storage in processing by the CPU; 108, a signalprocessing circuit which performs compression processing for recording;109, an interface for connection to a detachable recording medium; 110,an interface (e.g., a USB interface) for connection to a personalcomputer or the like; 111, a detachable recording medium (memory card);and 112, an operation unit which is comprised of buttons such as ashutter button and photographing/recording mode switch and a liquidcrystal display device.

FIG. 2 is a flowchart showing the step of storing the image data fromthe digital still camera into the recording medium 111. This processingis also performed when the shutter button of the operation unit 112 ispressed.

In step S201, the digital image data obtained by converting an analogsignal from the CCD 103 into digital data by using the A/D converter 104is stored in the internal memory 107. The A/D converter 104 convertseach component forming one pixel into digital data consisting of thenumber of bits larger than 8 bits (e.g., 12 bits) in order to improvethe precision of internal image processing (to be described later) andmeet demands from a skilled user in current processing (to be describedlater).

In step S202, it is determined whether or not image processing is to beperformed. This determination is performed in accordance with arecording mode set by the photographer with the operation unit 112 atthe time of photography. The recording modes include a raw imagerecording mode and a normal recording mode. The latter mode comes withthe size of an image to be recorded, image quality, sharpness andbrightness correction, and the like. In either mode,compression/non-compression can be set. If the execution of compressionin the raw image recording mode is set, lossless compression (e.g.,Huffman coding in TIFF or the like) is performed. If the execution ofcompression in the normal recording mode is set, JPEG Baseline coding(lossy compression) is performed.

If it is determined in step S202 that the normal recording mode isdesignated, the flow advances to step S203 to perform image processingin accordance with the conditions set with the operation unit 112. Inthis embodiment, filter processing, color adjustment based oninformation such as white balance information and autofocus information,and the like are performed. In this stage, each component forming apixel is rounded to 8 bits.

In step S204, it is determined whether or not compression recording isto be performed. This determination is performed on the basis ofinformation indicating necessity/unnecessity of compression which is setwith the operation unit 112.

If it is determined that compression is performed, the flow advances tostep S205. In the raw image recording mode, the raw image data stored inthe internal memory 107 is losslessly compressed. That is, eachcomponent obtained by the A/D converter 104 is losslessly compressedwith no reduction in the number of bits from 12 bits. In the normalrecording mode, JPEG Baseline coding (lossy) is performed. In JPEGBaseline coding, a component of each pixel is expressed by 8 bits.

The flow then advances to step S206 to generate attribute informationfrom photographing conditions and recording conditions such as aphotography date, an apparatus model for photography, flash on/off, anda recording format for a sensed image file, and add the information tothe image data. In step S207, the resultant data is written as a file inthe recording medium 111.

Note that a compressed file and uncompressed file can exist in therecording medium 111, and an image data file obtained by A/D-convertinga signal output from the CCD 103 and can be converted back into an image(each component consisting of 12 bits) without image processingassociated with color after white balance processing will be referred toas raw image data.

Note that the following may be referred to as raw image data: image datawhich is obtained by A/D-converting an image signal obtained from theimaging sensor and has not yet undergone color separation processing ofseparating the signal into a luminance signal and chrominance signals orimage data which is obtained from an output signal from a single-CCDsensor having color filters of a plurality of colors and has notundergone color interpolation processing. Alternatively, image datawhich is obtained from an image sensor having multicolor photosensitivelayers and has not yet undergone color plane data generation processingmay be referred to as raw image data.

FIG. 3 shows the structure of an image file stored in the recordingmedium 111.

Referring to FIG. 3, reference numeral 301 denotes an image attributeinformation area, which is used to store information at the time ofphotography, e.g., an image data size, photography date, apparatus modelfor photography, compression method, and flash on/off and informationnecessary for reading, reproduction, and selection of an image; 302, areduced image data area, which is used to store reduced image data usedfor the display of a list of image files; and 303, an image data area,which is used to store all image data to be stored as an image file.

FIGS. 4A to 4C show the array of the photoelectric conversion elementsof the CCD 103 according to this embodiment. This array is generallycalled a Beyer array. The data digitized by the A/D converter 104 andstored is divided into blocks like those shown in FIG. 4A when the CCDincludes primary color filters and a monochrome photosensitive layer. Asshown in FIG. 4B, one block is comprised of four components includingcolor signals of red, blue, and green (RGB). Such identical blocks arecontinuously and repeatedly arranged to form an array like the one shownin FIG. 4A. Raw image data in this embodiment is data having eachcomponent expressed by 12 bits.

When the data of three components R, G, and B are to be generated (JPEGcoding is to be performed) like one pixel of a normal color image, therespective R, G, and B data are calculated as follows.

Assume that a position (x, y) of interest coincides with the position ofthe primary color filter R, as shown in FIG. 4C. At the position (x, y),the data of the R component exists, but those of the B and G componentsdo not exist. The R, G, and B data are therefore obtained in thefollowing manner. Assume that in the following equations, D(x, y)represents the value of the coordinates (x, y) of raw image data.R(x, y)=D(x, y)G(x, y)={D(x+1, y)+D(x−1, y)+D(x, y+1)+D(x, y−1)}/4B(x, y)={D(x+1, y−1)+D(x−1, y+1)+D(x+1, y+1)+D(x−1, y−1)}/4  (1)In the above case, the coordinate position of interest coincides with anR primary color filter. Even if, however, the pixel position of interestcoincides with a G or B primary color filter, R, G, and B data areobtained in the same manner. Note, however, that if a G component is acomponent of interest, since there are only two sets of B and R dataadjacent to the G component, the above equations are slightly changed.

A raw image data file of the image files recorded on the recordingmedium 111 is read out by a general-purpose information processingapparatus such as a personal computer (to be referred to as a PChereinafter), and developing and editing processes are performed byapplication programs executed on the apparatus. This operation will bedescribed below.

FIG. 11 is a block diagram showing the arrangement of the PC in thisembodiment. Referring to FIG. 11, reference numeral 1 denotes a PC body;2, a CPU which controls the overall apparatus; 3, a ROM storing a BIOSand boot program; 4, a RAM; 5, a keyboard (KB) or a pointing device (PD)such as a mouse; 6, a display control unit incorporating a video memory;7, a display device (e.g., a CRT or LCD display) which displays a videosignal output from the display control unit; 9, a card reader whichaccesses a storage medium such as a memory card removed from the digitalcamera; and 10, an interface which connects to the interface 110 of adigital camera 20 (see FIG. 1).

In the above arrangement, when the power supply is turned on, the CPU 2loads the OS from a hard disk unit 8 into the RAM 4 in accordance withthe boot program in the ROM 3, and also loads an image editingapplication program according to this embodiment into the RAM 4 inaccordance with an instruction from the user. As a consequence, thisapparatus functions as an image processing apparatus. When theapplication in this embodiment starts, an image file is read and a workarea in which various kinds of variable are to be ensured and a cachearea are ensured, as shown in FIG. 11.

FIG. 12 shows a GUI for image editing which appears when the imageediting application program in this embodiment is started. As shown inFIG. 12, when the editing program in this embodiment is executed, twowindows 40 and 45 are displayed on the display screen.

The window 40 is used to mainly display an image in the process of beingedited, and is provided with an image display area 43 which displays animage as a result of a developing process or editing, a display sizeinstructing portion 41 which is used to designate the size of the imagedisplay area 43, a magnification instructing portion 50 which is used todesignate a display magnification, a save button 42 which is operated tostore an editing result, and a “file” button 44 which is operated todisplay a dialog box (not shown) for loading an image file. Upward anddownward arrows are displayed on the right end of the display sizeinstructing portion 41. The size of the image display area 43 can bechanged by moving a cursor interlocked with a mouse to one of the arrowsand performing clicking operation (to be simply referred to as operationhereinafter). The magnification instructing portion 50 is operated inthe same manner. The size of the image display area 43 may be set to belarger than that of the window 40. The size of the window 40 itself canbe changed by dragging a lower right portion 52 of the window 40.

Consider the magnification instructed by the magnification instructingportion 50. A magnification of 100% indicates that one pixel of anoriginal image (raw image) corresponds to one pixel of the image displayarea 43. Therefore, “20%” in FIG. 12 indicates that 5×5 pixels of theoriginal image correspond to one display pixel. That is, both thehorizontal and vertical sizes are ⅕ those of the original image.

When a portion of an original image is displayed in the image displayarea 43, an image portion at a different position can be seen byscrolling the image within the image display area 43. An image in theimage display area 43 is scrolled by moving the mouse cursor (not shown)into the image display area 43 and performing dragging operation (movingthe mouse while pressing its button). However, vertical and horizontalscroll bars may be prepared additionally to realize this operation byoperating them.

Consider the window 45 next. Various image editing items are displayedin this window. The user can adjust a desired editing item by moving thetab of a slider of the desired item (in FIG. 12, three levels, i.e.,“low”, “middle”, and “high”, are set, but as is obvious, more levels maybe set). Referring to FIG. 12, as editing items, three items, i.e., acontrast control 46, color density control 47, and sharpness control 48,are displayed. However, by operating a scroll bar 49, other settingitems (tone curve, brightness, white balance set value, colortemperature, hue, and the like) can be displayed. In a state (default)immediately after the application in this embodiment is started, the tabpositions of the slider bars of the respective editing items aredetermined by set information at the time of photography of thecorresponding image. That is, when an image file is loaded, parametersare extracted from attribute information 301 of the image, and the tabposition of the slider bars of the respective editing items aredetermined on the basis of the parameters. Note that if no correspondingset information exists in the attribute information 301, the tapposition of the slider bar of the corresponding editing item is set tobe located in the middle of the slider.

In the following description, the status value determined by the tabposition of a slider bar of each editing item will be referred to as anediting parameter (or simply a parameter), and the operation of changingthe state of a the editing item by changing its tab position will bereferred to as “setting a parameter”. Although the degree of editing foreach item can be set with high operability by using a slider bar asshown in FIG. 12, such setting may be made by inputting a numericalvalue. Any display form can be used for this setting operation. Althoughthe windows 40 and 45 are displayed, data may be displayed in onewindow.

The processing of displaying the image display area 43 in the executionof the application program according to the embodiment will be describedbelow with reference to the flowchart of FIG. 5. Assume that the userhas stored the image file obtained by image sensing by the digitalcamera in the hard disk unit 8 through the card reader 9 and interface10.

As is known, the number of pixels obtained by the CCD 103 of the recentdigital camera has increased, and has become much larger than the numberof pixels that can be displayed on the display device 7 of the PC. Evenas far as the OS for GUI is concerned, the size of a window that can beused by an application is inevitably smaller than the display area ofthe display device. It is therefore impossible to display an overallimage on the display device at once while making one pixel of the sensedimage data correspond to one pixel displayed on the display device (anenlargement ratio of 100%). For this reason, when a sensed image is tobe displayed at an enlargement ratio of 100%, it cannot help but todisplay a portion of the sensed image in the image display area 43. Inaddition, in order to display an overall sensed image, there is no waybut to make a plurality of pixels of the sensed image correspond to onedisplay pixel by reducing the enlargement ratio to 20% (0.2 times) ordecimate pixels at some proper intervals. The image editing applicationaccording to this embodiment is directed to perform an image editingprocess with focus being on such a point.

First of all, in step S501, a raw image file is loaded from, forexample, the hard disk of a personal computer (or through the cardreader 9 and interface 10) as an input means into the RAM 4.

In step S502, the attribute information 301 is loaded. This attributeinformation is imaging conditions determined when a raw image is sensed,and includes, for example, a bit depth (12 bits in this embodiment),compression format, exposure time, shutter speed, aperture value,luminance value, exposure correction value, lens minimum F value, objectdistance, photometric system, light source, flash on/off, lens focallength, F-number, exposure program, ISO speed rate, and flash intensity.

In step S503, the attribute information is analyzed to determine whetheror not the image data is compressed. If the image data is compressed,the data is decompressed (decoded) on the basis of the compressionformat indicated by the attribute information in step S504. As aconsequence, uncompressed raw image data can be obtained. Note, however,that at this point of time, each component is expressed by 12 bits.

In step S505, color interpolation processing is performed. The raw imageis comprised of data in blocks each shown in FIG. 4B. For this reason,color interpolation processing is performed. This interpolationprocessing is, for example, linear interpolation, and hence R, G, and Bdata are generated by equations (1) given above.

In step S506, it is determined whether or not the user has setinformation for image quality adjustment on the personal computer (thisset information will be referred to as user image quality adjustmentinformation hereinafter). User image quality adjustment informationincludes, for example, sharpness, contrast, color matrix, tone curve,lightness, saturation, hue, and white balance, which are set with thekeyboard or pointing device and temporarily stored in the RAM 4.

If user image quality adjustment information is set, the information isloaded from the RAM 4 in step S507.

In step S508, reproduction parameters for reproducing the raw image aregenerated from both the attribute information loaded in step S502 andthe user image quality adjustment information loaded in step S507, andwhite balance processing is performed on the basis of the generatedreproduction parameters. Attribute information necessary for whitebalance processing includes, for example, light source information,flash on/off, and flash intensity.

In step S509, R, G, and B primary color data (R, G, B) are convertedinto luminance and color difference data (Y, CbCr).

In step S510, luminance adjustment processing is performed on the basisof the attribute information loaded in step S502 described above and/orthe user image quality adjustment information loaded in step S507.Attribute information necessary for luminance adjustment processingincludes, for example, an exposure time, shutter speed, aperture value,luminance value, exposure correction value, object distance, photometricsystem, F-number, and exposure program.

In step S511, color adjustment processing is performed on the basis ofthe attribute information loaded in step S502 described above and/or theuser image quality adjustment information loaded in step S507.

In step S512, sharpness adjustment processing is performed on the basisof the attribute information loaded in step S502 and/or the user imagequality adjustment information loaded in step S507. Attributeinformation necessary for sharpness processing includes, for example, anexposure time, shutter speed, aperture value, luminance value, exposurecorrection value, object distance, photometric system, F-number,exposure program, lens distortion information, lens focal length, lensminimum F value, and ISO speed rate.

In step S513, luminance and color difference data (Y, Cb, Cr) areconverted into primary color data (R, G, B), and each component isrounded to 8 bits to display the data. In other words, since thecomputation precision is 12 bits in steps S505 to S513, computationerrors can be minimized.

In step S514, the resultant data is output to the display device 7. Thedisplay device 7 may be a CRT display, liquid crystal display, or thelike.

The processing in steps S508 to S513 is performed on a pixel basis, thetime required for this processing is proportional to the number ofpixels. Note that in this embodiment, the processing from the whitebalance processing in step S508 to the conversion of the data into RGBdata in step S513 after the color interpolation processing of the rawimage in step S507 will be referred to as a developing processhereinafter.

This embodiment has exemplified the processing using the primary colorfilters and monochrome photosensitive layer. This processing, however,may vary depending on imaging sensor filter characteristics. Forexample, when a complementary color filter comprised of color signals ofcyan, magenta, and yellow (CMY) is to be used or pixels of a pluralityof colors are arrayed in different orders, the aspect ratio of the shapeof a pixel is not 1:1 (e.g., 2:1). In each case, it suffices ifprocessing after step S505 corresponds to the filter characteristics ofeach imaging sensor.

The correspondence relationship between the size of the image displayarea 43, the display magnification, and the original image size in thedisplay device will be described next.

FIGS. 6A to 6C show examples of output results at differentmagnifications on the PC according to this embodiment.

Assuming that FIG. 6A shows the display output result of one-to-onedisplay of an overall raw image, i.e., display at a magnification of100%, FIGS. 6B and 6C respectively show the display output results at50% and 25%. The number of pixels of the image displayed at amagnification of 50% in each of the vertical and horizontal directionsis ½ that of the image displayed at a magnification of 100%. The totalnumber of pixels (area) of the image displayed at a magnification of 50%in each of the vertical and horizontal directions is the square of ½,i.e., ¼ that of the image displayed at a magnification of 100%. Thenumber of pixels of the image displayed at a magnification of 25% ineach of the vertical and horizontal direction is ¼ that of the imagedisplayed at a magnification of 100%. The total number of pixels (area)of the image displayed at 25% is the square of ¼, i.e., 1/16. That is,the number of pixels of an image displayed at a magnification of n% ineach of the vertical and horizontal directions is n/100 that of theimage displayed at a magnification of 100%, and the number of pixels(area) of the overall image is the square of n/100.

Decimation processing to be performed when the magnification of anoverall raw image subjected to target image extraction processing is setto n % in this embodiment will be described below.

An image with a pixel count (x) in the horizontal direction and a pixelcount (y) in the vertical direction is reduced to n/100 (n is a positiveinteger of 100 or less) by decimating the number of pixels in thehorizontal direction by (x×(100−n)/100) and decimating the number ofpixels in the vertical direction by (y×(100−n)/100). For example,decimation of pixels is performed by the following method. First of all,(n/100) is reduced to divisors expressed by (i/k). Then, i pixels aredecimated at equal intervals from k pixels. If reduction of fractioncannot be performed or pixels cannot be decimated at equal intervals, itsuffices to use a method which can reduce the number of pixels in thehorizontal direction to (x×n/100) and the number of pixels in thevertical direction to (y×n/100) with a small amount of processing andmost desirable image quality. In this manner, target image extractionprocessing is performed.

A more desirable reduction output result may be obtained by performingsmoothing processing before, during, or after decimation processing.Smoothing processing is image processing of smoothing an image havingundergone decimation of pixels to make it look more smoother. Forexample, output pixel information Out(x, y) in the image display area 43is calculated from a coordinate position (x, y) of an original image andits neighboring pixels according to equation (2):Out(x, y)={In(x, y)x2+In(x−1, y)+In(x+1, y)+In(x, y−1)+In(x,y+1)}÷6  (2)

FIGS. 7A to 7G show output results at the respective sizes of the imagedisplay area 43 which are set by the image processing apparatusaccording to this embodiment.

For example, the image surrounded by the dotted line indicates anoverall original image. When the size of the image display area 43 isset to that shown in FIG. 7B and the magnification is set to 100%, theimage within a frame 71 in FIG. 7C is displayed.

This will be described in more detail below. In the followingdescription, the unit of coordinates is pixels.

Let (0, 0) be the coordinates of the upper left corner of an originalimage 70, and (1536, 2048) be the coordinates of the lower right corner.Assume that the size of the image display area 43 is set to that inTable 1. Consider a case wherein the coordinates (250, 300) in theoriginal image are made to correspond to the upper left corner positionin the image display area 43, and the resultant image is displayed at amagnification of 100%. In this case, a pixel position (x1, y1) in theoriginal image which is to be displayed at the lower right corner in theimage display area 43 is given by equations (3).

TABLE 1 Magnification 100% Number of Pixels of Image Display 440 pixelsArea in Horizontal Direction Number of Pixels of Image Display 320pixels Area in Vertical Directionx1=250+(440×100÷100)−1y1=300+(320×100÷100)−1  (3)

When the size of the image display area 43 is increased from the size inFIG. 7B to the size in FIG. 7D with the same magnification, imageinformation in the solid line frame in FIG. 7E is displayed.

More specifically, when the size of the image display area 43 is set tothe size in table 2, and a pixel of the original image at thecoordinates (80, 120) is to be displayed at the upper left corner in theimage display area 43, the coordinates of the original image to bedisplayed at the coordinates (x1, y1) on the lower right corner in theimage display area are obtained by equations (4).

TABLE 2 Magnification 100% Number of Pixels of Image Display 800 pixelsArea in Horizontal Direction Number of Pixels of Image Display 560pixels Area in Vertical Directionx1=80+(800×100÷100)−1y1=120+(560×100÷100)−1  (4)

When the size of the image display area 43 is equal to that shown inFIG. 7D and the magnification is to be set to 200%, the image in a solidline frame 73 in FIG. 7G is displayed as shown in FIG. 7F.

More specifically, when a pixel of the original image at the coordinates(80, 120) is to be displayed at the upper left corner in the imagedisplay area 43, the coordinates of the original image to be displayedat the coordinates (x1, y1) on the lower right corner in the imagedisplay area are obtained by equations (5).

More specifically, when the size of the image display area 43 is set tothe size in table 3, and a pixel of the original image at thecoordinates (80, 120) is to be displayed at the upper left corner in theimage display area 43, the coordinates of the original image to bedisplayed at the coordinates (x1, y1) on the lower right corner in theimage display area are obtained by equations (5).

TABLE 3 Magnification 200% Number of Pixels of Image Display 800 pixelsArea in Horizontal Direction Number of Pixels of Image Display 560pixels Area in Vertical Directionx1=80+(800×100÷200)−1y1=120+(560×100÷200)−1  (5)

The above processing can be summarized as follows. Assume that the sizeof the image display area 43 is set to that shown in Table 4, themagnification (an integer indicated by a percentage) is M, and thecoordinates of the original image to be displayed at the upper leftcorner in the image display area 43 are given by (x0, y0). In this case,coordinates (x1, y1) of a portion in the original image which is to bedisplayed at the lower right corner in the image display area 43 can beobtained by equations (6).

TABLE 4 Magnification M % Number of Pixels of Image Display Window_Wpixels Area in Horizontal Direction Number of Pixels of Image DisplayWindow_H pixels Area in Vertical Directionx1=x0+(Window_(—) W×100÷M)−1y1=y0+(Window_(—) H×100÷M)−1  (6)

As is obvious from the above description, an extraction range (theposition and size of each of the frames 71 to 73) from the originalimage is determined from the size Window_W and height Window_H of theimage display area 43, the corresponding magnification M, and the pixelposition (x0, y0) of the original image to be displayed at the upperleft corner position in the image display area 43.

Although, therefore, a developing process needs to be performed for anoverall original image when the developing result is to be stored, adeveloping process during editing needs only to be performed for apartial area within an extraction range of the original image. Thismakes it possible to obtain a high response.

FIG. 8 is a flowchart showing image extraction processing (to bereferred to as a target image extraction processing hereinafter) to beperformed before a developing process. The processing in steps S801 toS807 will be referred to as target image extraction processinghereinafter.

This processing will be described with reference to the flowchart ofFIG. 8. The respective variables in the description have the followingmeanings:

-   fulpix_w: the number of pixels of a raw image in the horizontal    direction-   fulpix_h: the number of pixels of the raw image in the vertical    direction-   window_w: the number of pixels of the image display area 43 in the    horizontal direction-   window_h: the number of pixels of the image display area 43 in the    vertical direction-   M: the magnification (%) designated by the magnification instructing    portion 50-   C_w: the number of pixels of the original image based on the    magnification M in the horizontal direction-   C_h: the number of pixels of the original image based on the    magnification M in the vertical direction

In step S801, the image information of an overall image as a processingtarget is loaded from the raw image.

In step S802, information associated with display is loaded. Theinformation associated with display includes the size informationwindow_w and window_h of the image display area 43 displayed on themonitor and the magnification information M.

In step S803, the raw pixel sizes C_w and C_h based on the magnificationare calculated according to equations (7):C _(—) w=(fulpix_(—) w×M÷100)C _(—) h=(fulpix_(—) h×M÷100)  (7)

In step S804, the size window_w/window_h of the image display area 43 iscompared with the size C_w/C_h of the raw image in which themagnification M is reflected. More specifically, it is determinedwhether or not inequalities (8) and (9) are true:window_w<C_w  (8)window_h<C_h  (9)

If it is determined in step S804 that inequalities (8) and (9) aresatisfied, it means that the overall raw image cannot be displayed. Thatis, a portion of the raw image is to be displayed. In step S805,therefore, the size and position of an extraction area for partialextraction of an image is calculated. The size of this extraction areamay be obtained by equations (6) given above. Note that in equations(6), the coordinates (x0, y0) may be obtained in the following manner,when a developing process is performed for the raw image data for thefirst time, and the central position of the image display area 43 ismade to coincide with the central portion of the raw image data:x0=fulpix_(—) w/2−window_(—) w/2y0=fulpix_(—) h/2−window_(—) h/2

If it is determined in step S804 that inequalities (8) and (9) are notsatisfied, the size (width, height) of the overall image correspondingto the magnification M falls within the size of the image display area43, and the overall raw image is displayed. The flow therefore advancesto step S806.

In step S806, it is determined whether or not the magnification M islower than 100% (the size of the original raw image), i.e., “M<100” issatisfied. If the magnification M is lower than 100%, since the overallimage is reduced, the image data in the extraction area in the raw imagedata is decimated in accordance with “M/100”, and the result is outputas the processing result on “target image extraction processing” in thisembodiment. This decimation processing is performed by the same methodas that described with reference to FIGS. 6A to 6C. If the magnificationM is equal to or more than 100% (one pixel of raw image=one displaypixel), no decimation processing is performed, and the image data in theextraction area determined in step S805 is output as the result on“target image extraction processing”.

In the above manner, image data is extracted from an extraction area inraw image data. For this extracted image data, the developing process insteps S508 to S513 in FIG. 5 is performed. Performing extractionprocessing in accordance with a display state in this manner can make animage as a target for a developing (reproduction) process become aportion of the raw image which is determined by the size of the imagedisplay area 43 and a set magnification. As a consequence, the number ofpixels subjected to a developing process can be decreased, and hence theprocessing time required for the developing process can be shortened.

In addition, the raw image which has undergone color interpolation instep S505 may be held in a cache memory or the like. In this case, sincethe time required for color interpolation processing can be saved, theprocessing speed can be further increased.

Target image extraction processing in this embodiment can shorten thedeveloping processing time as compared with a conventional developingprocess. This point will be described below. Prior to the description,variables are defined as follows:

-   bef_time: the processing time in the prior art (i.e., the processing    time required to perform a developing process for overall image    information)-   aft_time: the processing time required when this embodiment is used-   pre_time: the time require for the processing in steps S501 to S505,    S506, and S507-   dev_time: the time taken for a developing process per pixel-   out_time: the time taken for the processing in step S514-   ext_time: the time taken for target image extraction processing-   outpix_w: the number of pixels of an image in the horizontal    direction after target image extraction processing-   outpix_h: the number of pixels of an image in the vertical direction    after target image extraction processing-   vis_width: the number of pixels in the horizontal direction in a    display area which corresponds to a magnification-   vis_height: the number of pixels in the vertical direction in a    display area which corresponds to a magnification

The processing (bef_time) in the prior art is given:bef_time=pre_time+dev_time×(fulpix_(—) w×fulpix_(—) h)+out_time  (10)

In contrast to this, the processing time (aft_time) taken when targetimage extraction processing according to the present invention isperformed is given byaft_time=pre_time+ext_time+dev_time×(outpix_(—) w×outpix_(—)h)+out_time  (11)

Obviously, the values of fulpix_w, fulpix_h, outpix_x, and outpix_h aredominant for the times calculated by equations (10) and (11).

The following is a method of calculating the numbers of pixels(outpix_w, outpix_h) after target image extraction processing. Assumethat the size (fulpix_w, fulpix_h) of the original image and information(M, window_w, window_h) associated with display have already been loaded(assigned).

C_h = fulpix_w*M/100; C_w = fulpix_h*M/100; if(C_W > window_w){vis_width = window_w; }else{ vis_width = C_w; } if(C_h > window_h){vis_height = window_h; }else{ vis_height = C_h; } outpix_w = vis_width*100/M; outpix_h = vis_height*100/M; ...(12)

In this embodiment, equation (13) given below is always true.outpix_(—) w×outpix_(—) h≦fulpix_(—) w×fulpix_(—) h  (13)

If, therefore, inequality (14) is satisfied, inequality (15) is true.

ext_time < dev_time × ((fulpix_w × fulpix_h) − (outpix_w × outpix_h))...(14) bef_time > aft_time ...(15)

This embodiment has exemplified information recorded and attached to afile as attribute information. However, attribute information can berecorded on another file or stored in a different medium as long as itis associated with image information and recorded under a condition thatallows the image processing apparatus to load it.

In this embodiment, when the user has performed only image qualityadjustment without changing the image display area 43 and magnificationM (i.e., has changed only user image quality adjustment information),the processing time can be further shortened by performing theprocessing in steps S508 to S514 for image information after the targetpixel extraction, as long as the image information after the previoustarget pixel extraction processing is held in a storage medium such as acache memory. Note, however, that this can be realized on the conditionthat the time required to load information from a storage medium(read_time) is shorter than the time until target image extractionprocessing (pre_time+ext_time). Equation (16) represents the processingtime (cash_time) required when the processing in steps S508 to S514 isperformed for image information after target image extractionprocessing.cash_time=read_time+dev_time×(outpix_(—) w×outpix_(—) h)+out_time  (16)

If expression (17) holds, expression (18) is true.read_time<pre_time+ext_time+dev_time×((fulpix_(—) w×fulpix_(—)h)−(outpix_(—) w×outpix_(—) h))  (17)bef_time>cash_time  (18)

In this embodiment, decimation processing (step S807) is performed aftera range subjected to image reproduction and processing is calculated(step S805). However, the same effect as described above can be obtainedeven if the range is calculated after the decimation processing isperformed.

If color interpolation processing, decimation processing, and extractionprocessing for a portion of the display area can be done simultaneouslyor concurrently, they may be done simultaneously or concurrently.Obviously, holding image information after color interpolationprocessing in a cache memory or the like will further speed up targetimage extraction processing, a developing process based on reproductionparameters set thereafter by the user, and display such as scrolling.

This embodiment has exemplified the case wherein target image extractionprocessing is performed for the image obtained by color interpolationprocessing for a raw image. Alternatively, if index information such asmarking is inserted at intervals of a predetermined number of pixels atthe time of recording a losslessly compressed raw image so as to allowrecognition of a predetermined pixel position from the compressed rawimage, target image extraction processing may be performed before thedecompression processing in step S504. That is, after target imageextraction processing is performed with respect to the raw image,decompression processing is performed for only the extracted range, andcolor interpolation and a subsequent developing process are performed.Performing decompression processing for only an extracted target imagein this manner will eliminate the necessity to decompress the overallimage in advance. This makes it possible to further shorten pre_timedescribed above. In this case, although decompression processing andcolor interpolation processing after target image extraction areperformed before a developing process, since the time required for thisdecompression processing is shorter than that required for thedecompression of the overall image, the image after the developingprocess can be displayed at high speed.

In this embodiment, equations (6) represent the method of calculating arange as a target for image reproduction and a developing process. Ifneighboring pixels of an output pixel are required in a developingprocess as in the case of smoothing processing represented by equation(2), it suffices to perform a developing process for pixels in a rangelarger than the above range by the required numbers of pixels in thehorizontal and vertical directions as target pixels and display only acorresponding range in the display area at the time of display.

In this embodiment, if the size of the image display area 43 is smallerthan the size of a raw image and hence is not suitable for a check onimage quality associated with resolution, e.g., is 100% or less, someproper processing in a developing process used in the case of 100% maybe omitted to increase the processing speed as long as proper imagequality as an index for image quality adjustment can be maintained. If,for example, the magnification is set to 25%, since the resultant imageis difficult to use as a criterion for checking image quality associatedwith resolution, sharpness processing may be simplified. If onlysharpness processing is to be performed, it suffices to only display animage extracted in accordance with the magnification without reflectingthe processing result in the image. In this case, sharpness processingmeans aperture correction processing or edge enhancement processing.

As has been described in the first embodiment, a developing processresult can be displayed at a higher speed than in the prior art bychanging image information as a target for processing in accordancewith, for example, an arbitrary magnification and the size of an imagedisplay area.

Note that the operator operates the save button 42 when he/she finallysatisfies a developing process result in the above manner. When the savebutton 42 is operated, the editing result on the raw image data ofinterest is stored as a file according to the sequence shown in FIG. 13.

More specifically, in step S1301, raw image data is loaded. In stepS1302, a developing process is performed for the input raw image inaccordance with the parameters set on the window 45 immediately beforethe save button 42 is operated. This developing process may be performedon the assumption that the magnification is 100% and the size of theimage display area 43 is equal to the size of the raw image. In stepS1303, the processing result is stored as a file in the hard disk unit 8or the like.

Second Embodiment

The first embodiment has exemplified how the display of an image after adeveloping process for real display is speeded up. It is preferable toperform the same image processing in the following cases: when imagequality adjustment is performed (parameters for contrast, lightness, andthe like are changed), when the size of an image display area 43 ischanged, and when an image in the image display area is scrolled. If,however, a developing process is performed every time an image isscrolled by a slight distance while a final scroll position is notconfirmed, for example, the image in the image display area 43 isscrolled, the scrolling operation becomes awkward. That is, in scrollprocessing, it is preferable to quickly respond to scrolling operation.This also applies to a case wherein the size of the image display area43 is changed.

The second embodiment, therefore, will exemplify a case wherein whilethe size of the image display area 43 is changed or an image in theimage display area 43 is scrolled, a simplified image is displayed untilan image adjustment result is obtained and a final processing result isobtained. For the sake of simple explanation, it will described that asimplified image is displayed until an image adjustment result isobtained.

FIG. 9 is a flowchart showing a sequence of displaying a low-resolutionimage until an image processing result corresponding to informationassociated with display (an image quality adjustment parameter, the sizeof the image display area 43, and the magnification) is output.

The processing from step S901 to step S905 is the same as that from stepS501 to step S505 in the first embodiment, and hence a descriptionthereof will be omitted.

In step S906, low-resolution pixels for temporary display are extracted.The extraction method is the same as that used when decimationprocessing with a magnification of 100% or less is performed for anoverall image.

The resolution (magnification) of image information for temporarydisplay is set and registered in advance by an operator on the basisthat proper image quality for temporary display is ensured and theoperator can feel that the processing time is short.

The image information for temporary display can be used when imagequality adjustment is performed or display information is changed aswill be described later with reference to FIG. 10, and hence is held ina proper area in the RAM 4. However, this information may be stored in ahard disk unit 8.

In step S907, target pixel extraction processing is performed. The imageinformation after this target pixel extraction processing is held in acache area, as described in the first embodiment. If the imageinformation after the target pixel extraction processing cannot bereused as in a case wherein information associated with display ischanged, the image information is discarded.

The processing in steps S908 and 909 is the same as that in steps S506and S507 in the first embodiment.

In step S910, a developing process is performed for the imageinformation for temporary display. In step S911, the image processingresult obtained in step S910 is output to the image display area 43 of adisplay device 7. This processing is the same as that in step S514. As aresult, a simplified image is displayed in the image display area 43until a true developing process is complete.

In step S912, a true developing process is performed for the imageinformation after the target pixel extraction processing. In step S913,the image processing result obtained in step S912 is output to an outputapparatus. This processing is also the same as that in step S514.

FIG. 10 is a flowchart showing the processing to be executed when theuser performs again image quality adjustment for the image havingundergone the processing in FIG. 9.

First of all, in step S1001, user image quality adjustment informationset by the operator through a window 45 is loaded. This processing isthe same as that in step S507.

It is determined in step S1002 whether or not the user image qualityadjustment information has been changed. If it is determined in stepS1002 that the information has been changed, the processing from stepS1003 to step S1008 is performed.

In step S1003, it is determined whether or not the changed user imagequality adjustment information is a reproduction parameter foradjustment associated with color or brightness. The reproductionparameter for adjustment associated with color or brightness includes aparameter for contrast, color matrix, tone curve, lightness, saturation,hue, or white balance. If it is determined in step S1003 that adjustmentinformation associated with color and/or luminance has been changed, theprocessing from step S1004 to step S1006 is performed.

In step S1004, the image information for temporary display stored in theRAM in advance in step S906 is loaded. If there is no image informationfor temporary display (the size and position of an image extraction areahave been changed by scrolling or the like), the processing from stepS901 to step S906 is performed again to generate image information.

In step S1005, a developing process is performed for low-resolutionimage information for temporary display. This processing is the same asthat in step S910. In step S1006, the processing result on the image fortemporary display which is obtained in step S1005 is output to the imagedisplay area 43 to be displayed (as in the processing in steps S514 andS911).

In step S1007, the image information after the target pixel extractionprocessing which is stored in for example, the RAM is loaded. If thereis no image information after target pixel extraction processing, theprocessing in steps S901 to S905 and S907 is performed again to generateimage information.

In step S1008, a true developing process is performed for the imageinformation after the target pixel extraction processing (as in theprocessing in step S912).

In step S1009, the image processing result obtained in step S1008 isoutput to the image display area 43 to be displayed (as in theprocessing in steps S514 and S913).

Some proper processing in a developing process associated with imageinformation for temporary display in step S912 may be omitted as long asproper image quality for temporary display can be maintained. Forexample, sharpness processing, smoothing processing, noise reductionprocessing, and the like are difficult to use as targets for imagequality checks with respect to a low-resolution image such as an imagefor temporary display, and hence may be omitted or simplified.

In this embodiment, when information associated with display is changed,for example, an image in the display area is scrolled or the size of theimage display area 43 is changed, image information for temporarydisplay within a range corresponding to the display area may be held inthe cache memory, and the image for temporary display may be displayeduntil display switching is complete. In addition, if a reduced image fortemporary display which has been processed on the basis of developingparameters set by the user in step S910 or S1005 is held in the cachememory, the reduced image can be displayed without the execution of theprocessing in step S1004 or S1005 until the raw image is processed. Thismakes it possible to reduce the time lag that occurs when the displayimage after a developing process is switched, thereby allowing smoothdisplay update.

In this embodiment, image information for temporary display is extractedby decimating data from an overall image. However, such information maybe extracted by decimating only data within a range corresponding to thedisplay area as in target pixel extraction processing.

In this embodiment, it is determined in accordance with user imagequality adjustment information whether or not a processing result on animage for temporary display is to displayed. In addition to the abovedetermination condition, however, if the number of pixels after targetpixel extraction processing is smaller than the number of pixels fortemporary display or there is hardly any difference between them, it maybe determined that the processing result on the image for temporarydisplay is not to be displayed. As for a criterion for determining thatthere is hardly any difference, whether or not a processing speeddifference can be expected may be determined from the number of pixelsor the loading speed of image information.

Assume that in this embodiment, during the processing in step S1008,operation is performed with respect to the window 45, the size of theimage display area 43 is changed, or an image in the image display areais scrolled. In this case, if the processing in step S1008 isinterrupted and the flow returns to step S1001 to redo the processing,an image for temporary display can always be updated during coloradjustment and/or luminance adjustment. For example, there can beprovided an image apparatus designed to update the display of an imagefor temporary display in real time during operation of a slider bar forthe adjustment of contrast.

In this embodiment, the processing in steps S1004 and S1005 and theprocessing in steps S1007 and S1008 may be concurrently performed if theoperating environment allows concurrent processing. In the above case,if the processing in steps S1005 to S1008 is terminated earlier, theprocessing in step S1009 may be performed without the execution of theprocessing in step S1006.

As described above, according to the second embodiment, there can beprovided an image processing apparatus, in which when the size of theimage display area 43 is changed, or while an image is scrolled, or whenimage quality adjustment is performed on the window 45, a simplifiedimage for temporary display is displayed in the image display area 43until final image processing is complete, thereby performing displayupdate more smoothly than the first embodiment. In addition, there canbe provided an image processing apparatus which can perform a detailedcomparison by not performing temporary display when, for example,sharpness or smoothing processing is performed.

In this embodiment, data which is obtained by A/D-converting a signaloutput from the imaging sensor and has not undergone compressionprocessing is used as raw image. However, data which has undergonelossless compression after A/D conversion may be used as raw image.

In addition, a raw image may be an output analog signal obtained fromthe imaging sensor, an image which has undergone A/D conversion but hasnot undergone at least white balance processing, an image signal whichis obtained from the imaging sensor, has undergone A/D conversion, andhas not undergone color separation processing of separating the signalinto luminance and chrominance signals, or an output signal from a colorfilter such as a color filter having a Beyer array and has not undergonecolor interpolation processing.

In this embodiment, the CCD has been described as an imaging sensor. Thesame effects as described above can be obtained even if a CMOS sensor isused. When a sensor having multicolor photosensitive layers is used, thecolor data of each pixel is obtained instead of interpolationprocessing, but other operations are the same as those described above.

In the first and second embodiments, a developing process has beendescribed as the processing of outputting image data to a monitor suchas a CRT. However, this process includes the processing of printing andreproducing an image from a raw image on the basis of reproductionparameters.

In the embodiments, each component of raw image data is comprised of 12bits. However, the present invention is not limited to this number ofbits, and can be applied to any number of bits as long as it exceeds 8bits as in JPEG Baseline coding.

In addition, the processing described in each embodiment can beimplemented by application programs executed by a general-purposeinformation processing apparatus such as a personal computer which hashardware (e.g., a USB interface which a digital camera generally has ora device which accesses a storage medium such as a memory card removedfrom the digital camera) for inputting an image obtained by imagesensing by an image sensing device such as a digital camera. Obviously,therefore, the present invention incorporates such computer programs. Ingeneral, the compute programs are stored in a computer-readable storagemedium such as a CD-ROM, and are set in the computer to be copied orinstalled in the system, thereby allowing the programs to be executed.Obviously, therefore, the present invention incorporates such acomputer-readable storage medium.

As has been described above, according to the present invention, rawimage data sensed by a digital camera or the like can be processed withhigh response during a developing or editing process.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the claims.

CLAIM OF PRIORITY

This application claims priority from Japanese Patent Application No.2004-031401 filed on Feb. 6, 2004 and Japanese Patent Application No.2005-000910 filed on Jan. 5, 2005, the entire contents of which arehereby incorporated by reference herein.

1. An image processing apparatus which develops RAW image data obtainedby an image sensing apparatus having an imaging sensor, comprising: amagnification setting device that sets, in accordance with a user'soperation, a magnification representing a ratio of a number of pixelswithin a display region of a display device to a number of pixels in acorresponding area in the RAW image data; a read out device that readsout RAW image data; a developing device that develops RAW image data; anextracting device that extracts, from the RAW image data, a number ofpixels less than a number of all of pixels included in the RAW imagedata in accordance with the magnification set by said magnificationsetting device, where the lesser the magnification setting device setsthe magnification, the lesser ratio at which said extracting deviceextracts pixels if the magnification is less than 100%; a control devicethat causes said developing device to develop the pixels extracted bysaid extracting device; and a display control device that controls thedisplay device to display, within the display region, the image datadeveloped by said developing device.
 2. The apparatus according to claim1, further comprising a recording control device that causes, inaccordance with a user's operation, said developing device to developall of pixels included in the RAW image data, and records image dataobtained by the developing process onto a recording medium.
 3. Theapparatus according to claim 1, further comprising a parameter settingdevice that sets, in accordance with a user's operation, a parameter fora developing process, wherein said developing device executes thedeveloping process in accordance with the parameter set by saidparameter setting device.
 4. The apparatus according to claim 3,wherein, when said parameter setting device sets the parameter, saidcontrol device controls said developing device so as to develop pixelsextracted by said extracting device in accordance with the parameter. 5.The apparatus according to claim 3, wherein the parameter to be set bysaid parameter setting device includes a parameter for sharpnessprocessing, wherein, when the magnification is less than 100%, saiddeveloping device executes a sharpness process simpler than a sharpnessprocess for a magnification larger than 100% or does not execute thesharpness process.
 6. The apparatus according to claim 1, furthercomprising a generating device that generates small RAW image data, fortemporal display, having a small number of pixels by thinning out theRAW image data at a predetermined thinning ratio; and a holding devicethat holds the small RAW image data generated by said generating device,wherein, when the pixels extracted by said extracting device aredeveloped by said developing device, said control device controls saiddeveloping device so as to execute a developing process on the small RAWimage data held by said holding device as a preliminary imageprocessing, and controls the display device so as to display, within thedisplay region, an image obtained by the preliminary image processinguntil the developing process for the pixels extracted by said extractingdevice is completed.
 7. The apparatus according to claim 6, wherein, inthe preliminary image processing, said developing device executes thedeveloping processing omitting at least one of a sharpness process,smoothing process and noise reduction process.
 8. The apparatusaccording to claim 6, wherein, when a specific parameter amongparameters for developing is changed, said control device controls saiddeveloping device not to execute the preliminary image processing. 9.The apparatus according to claim 8, wherein the specific parameterexcludes parameters for contrast, color matrix, tone curve, brightness,saturation, hue and white balance.
 10. The apparatus according to claim6, wherein said generating device generates the small RAW image datafrom the whole of the RAW image data.
 11. The apparatus according toclaim 6, wherein said generating device generates the small RAW imagedata from pixels in an area, to be displayed within the display region,in the RAW image data.
 12. A method of controlling an image processingapparatus which develops RAW image data obtained by an image sensingapparatus having an imaging sensor, comprising the steps of: amagnification setting step of setting, in accordance with a user'soperation, a magnification representing a ratio of a number of pixelswithin a display region of a display device to a number of pixels in acorresponding area in the RAW image data; a read out step of reading outRAW image data; a developing step of developing RAW image data; anextracting step of extracting, from the RAW image data, a number ofpixels less than a number of all of pixels included in the RAW imagedata in accordance with the magnification set in said magnificationsetting step, where the lesser the magnification setting step sets themagnification, the lesser ratio at which said extracting step extractspixels if the magnification is less than 100%; a control step of causingsaid developing step to develop the pixels extracted in said extractingstep; and a display control step of controlling the display device todisplay, within the display region, the image data developed in saiddeveloping step.
 13. A non-transitory computer-readable storage mediumstoring a computer program to execute the steps defined in claim
 12. 14.An image processing apparatus which develops RAW image data obtained byan image sensing apparatus having an imaging sensor, comprising: aregion size setting device that sets, in accordance with a user'soperation, the size of a display region, of a display device, fordisplaying an image; a read out device that reads out RAW image data; adeveloping device that develops RAW image data; an extracting devicethat calculates, from the size of the display region set by said regionsize setting device, the size of a corresponding partial area in the RAWimage data in accordance with a user's operation, and extracts pixelsfrom the corresponding partial area having the calculated size, wherethe number of pixels to be extracted is less than a number of all ofpixels included in the RAW image data; a control device that causes saiddeveloping device to develop the extracted pixels included in thepartial area; and a display control device that controls the displaydevice to display, within the display region, an image developed by saiddeveloping device under control of said control device.
 15. Theapparatus according to claim 14, wherein, in response to the change ofthe size of the display region by said region size setting device, saidextracting device executes the extraction process and said controldevice controls said developing device to develop the extracted pixelsin the partial area.
 16. The apparatus according to claim 14, furthercomprising a magnification setting device that sets, in accordance witha user's operation, a magnification representing a ratio of a number ofpixels within the display region of the display device to a number ofpixels in a corresponding area in the RAW image data, wherein, if themagnification is lesser than 100%, the lesser the magnification settingdevice sets the magnification, the lesser ratio at which said extractingdevice extracts pixels from the RAW image data.
 17. The apparatusaccording to claim 14, further comprising an area changing device thatchanges, in accordance with a user's operation, the display area, in theRAW image data, to be displayed on the display region.
 18. The apparatusaccording to claim 17, wherein, in response to the change of the displayarea by said area changing device, said extracting device executes theextraction process and said control device controls said developingdevice to develop pixels in the partial area.
 19. The apparatusaccording to claim 14, further comprising a recording control devicethat causes, in accordance with a user's operation, said developingdevice to develop all of pixels included in the RAW image data, andrecords image data obtained by the developing process onto a recordingmedium.
 20. The apparatus according to claim 14, further comprising aparameter setting device that sets, in accordance with a user'soperation, a parameter for a developing process, wherein said developingdevice executes the developing process in accordance with the parameterset by said parameter setting device.
 21. The apparatus according toclaim 20, wherein, when said parameter setting device sets theparameter, said control device controls said developing device so as todevelop pixels in the partial area in accordance with the parameter. 22.The apparatus according to claim 20, wherein the parameter to be set bysaid parameter setting device includes a parameter for sharpnessprocessing, wherein, when magnification is less than 100%, saiddeveloping device executes a sharpness process simpler than a sharpnessprocess for a magnification larger than 100% or does not execute thesharpness process.
 23. The apparatus according to claim 14, furthercomprising a generating device that generates small RAW image data fortemporal display having a small number of pixels by thinning out the RAWimage data at a predetermined thinning ratio; and a holding device thatholds the small RAW image data generated by said generating device,wherein, when the pixels in the partial area extracted by saidextracting device are developed by said developing device, said controldevice controls said developing device so as to execute a developingprocess on the small RAW image data held by said holding device as apreliminary image processing, and controls the display device so as todisplay an image obtained by the preliminary image processing within thedisplay region until the developing process for the pixels in thepartial area extracted by said extracting device is completed.
 24. Theapparatus according to claim 23, wherein, in the preliminary imageprocessing, said developing device executes the developing processingomitting at least one of a sharpness process, smoothing process andnoise reduction process.
 25. The apparatus according to claim 23,wherein, when a specific parameter among parameters for developing ischanged, said control device controls said developing device not toexecute the preliminary image processing.
 26. The apparatus according toclaim 25, wherein the specific parameter excludes parameters forcontrast, color matrix, tone curve, brightness, saturation, hue andwhite balance.
 27. The apparatus according to claim 23, wherein saidgenerating device generates the small RAW image data from the whole ofthe RAW image data.
 28. The apparatus according to claim 23, whereinsaid generating device generates the small RAW image data from pixels inan area, to be displayed within the display region, in the RAW imagedata.
 29. A method of controlling an image processing apparatus whichdevelops RAW image data obtained by an image sensing apparatus having animaging sensor, comprising the steps of: a region size setting step ofsetting, in accordance with a user's operation, the size of a displayregion, of a display device, for displaying an image; a read out step ofreading out RAW image data; a developing step of developing RAW imagedata; an extracting step of calculating, from the size of the displayregion set by said region size setting step, the size of a correspondingpartial area in the RAW image data in accordance with a user'soperation, and extracting pixels from the corresponding partial areahaving the calculated size, where the number of pixels to be extractedis less than a number of all of pixels included in the RAW image data; acontrol step of causing said developing step to develop the extractedpixels included in the partial area; and a display control step ofcontrolling the display device to display, within the display region, animage developed in said developing step under control of said controlstep.
 30. A non-transitory computer-readable storage medium storing acomputer program to execute the steps defined in claim 29.